Introduce array_slots macro for type-safe array slot access - #8912
Conversation
Benchmarks: Vortex queries 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.995x ➖, 0↑ 0↓)
datafusion / parquet (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.976x ➖, 0↑ 0↓)
duckdb / parquet (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Merging this PR will not alter performance
|
Migrate every array encoding with a fixed set of child slots to the #[array_slots] proc macro, replacing hand-written slot index constants (*_SLOT/NUM_SLOTS/SLOT_NAMES), from_slots/into_slots plumbing, and per-slot ext-trait accessors with the generated Slots struct, SlotsView, and *ArraySlotsExt trait. Newly ported arrays: - vortex-array: Primitive, Bool, Decimal, VarBin, VarBinView, FixedSizeList, List, ListView, Filter, Slice, Extension, Shared, Variant - fastlanes: FoR, Delta, RLE - encodings: RunEnd, ZigZag, FSST, ALPRD, Pco, ByteBool, Zstd, DecimalByteParts, ParquetVariant Slot field order everywhere matches the previous slot indices, and slot names match the previous SLOT_NAMES, so serialization is unchanged. Validity-style slots are Option<ArrayRef>; Validity-returning helpers that would collide with generated accessors follow the Masked pattern (e.g. bytebool_validity, parquet_variant_validity, unsliced_validity). Ext traits that kept non-slot methods are now supertraited on the generated *ArraySlotsExt; traits that only had slot accessors were removed. Downstream call sites across vortex-cuda, vortex-arrow, vortex-btrblocks, vortex-compressor, vortex-duckdb, vortex-row, vortex-geo, vortex-tensor, fuzz, and benches import the generated traits. Arrays with variable-length children (Chunked, Struct, Union, Interleave, ScalarFn, ZstdBuffers) and zero-slot arrays (Null, Constant, Sequence) are intentionally not migrated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BGk7kM4exZVJ5STJxt4dxK Signed-off-by: Claude <noreply@anthropic.com>
50971c5 to
bb63034
Compare
…-macro-migration-btzeou Signed-off-by: Claude <noreply@anthropic.com> # Conflicts: # vortex-array/src/arrays/masked/execute.rs # vortex-compressor/src/compressor/cascade.rs
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.015x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.015x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.000x ➖, 1↑ 1↓)
datafusion / vortex-compact (0.999x ➖, 0↑ 1↓)
datafusion / parquet (0.987x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.990x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.043x ➖, 0↑ 2↓)
datafusion / vortex-compact (1.025x ➖, 0↑ 0↓)
datafusion / parquet (1.017x ➖, 1↑ 0↓)
datafusion / arrow (1.012x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.009x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.019x ➖, 0↑ 0↓)
duckdb / parquet (1.011x ➖, 0↑ 2↓)
duckdb / duckdb (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.064x ➖, 0↑ 21↓)
datafusion / vortex-compact (1.047x ➖, 0↑ 7↓)
datafusion / parquet (1.054x ➖, 0↑ 16↓)
duckdb / vortex-file-compressed (1.051x ➖, 0↑ 11↓)
duckdb / vortex-compact (1.037x ➖, 0↑ 7↓)
duckdb / parquet (1.028x ➖, 0↑ 1↓)
duckdb / duckdb (1.033x ➖, 1↑ 6↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.993x ➖, 1↑ 2↓)
datafusion / vortex-compact (0.968x ➖, 0↑ 0↓)
datafusion / parquet (1.018x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.112x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.102x ➖, 0↑ 0↓)
duckdb / parquet (1.032x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.012x ➖, 1↑ 2↓)
datafusion / parquet (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.007x ➖, 0↑ 1↓)
duckdb / parquet (0.986x ➖, 1↑ 0↓)
duckdb / duckdb (0.993x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 97↑ 104↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.851x ✅, 9↑ 0↓)
duckdb / vortex-compact (0.891x ✅, 7↑ 0↓)
duckdb / parquet (0.905x ➖, 4↑ 0↓)
No file size changes detected. |
Benchmarks: Random Access 📖Vortex (geomean): 0.952x ➖ How to read Verdict and Engines
unknown / unknown (0.959x ➖, 7↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.035x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.049x ➖, 0↑ 1↓)
datafusion / parquet (0.972x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.044x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.043x ➖, 0↑ 0↓)
duckdb / parquet (1.040x ➖, 0↑ 0↓)
|
…-macro-migration-btzeou
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.994x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.992x ➖, 0↑ 0↓)
datafusion / parquet (1.000x ➖, 0↑ 0↓)
datafusion / arrow (1.072x ➖, 0↑ 8↓)
duckdb / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (0.991x ➖, 0↑ 0↓)
duckdb / duckdb (1.026x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.020x ➖, 0↑ 0↓)
datafusion / parquet (1.011x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.995x ➖, 2↑ 1↓)
duckdb / parquet (1.005x ➖, 0↑ 0↓)
duckdb / duckdb (1.008x ➖, 0↑ 1↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: Appian on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.072x ➖, 0↑ 3↓)
datafusion / parquet (1.074x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed (1.002x ➖, 0↑ 0↓)
duckdb / parquet (1.000x ➖, 0↑ 0↓)
duckdb / duckdb (1.000x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.979x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.932x ➖, 0↑ 0↓)
datafusion / parquet (1.051x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.971x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.985x ➖, 0↑ 0↓)
duckdb / parquet (1.035x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Vortex (geomean): 0.999x ➖ How to read Verdict and Engines
unknown / unknown (0.994x ➖, 0↑ 0↓)
|
|
@codex can you review this pr, lookinng for any behaviour changes other than using auto gen slots |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Rationale for this change
This PR introduces a new
#[array_slots]procedural macro that provides type-safe, ergonomic access to array child slots. Previously, array implementations used manual constant definitions (e.g.,VALIDITY_SLOT = 0,METADATA_SLOT = 1) and error-prone index-based slot access viaslots[idx]. This approach was repetitive, error-prone, and lacked compile-time safety.What changes are included in this PR?
ParquetVariantArray(parquet-variant encoding)FSSTArray(fsst encoding)ALPRDArray(alp encoding)ListArray,ListViewArray,FixedSizeListArray(core arrays)VarBinArray,VarBinViewArray(core arrays)BoolArray,DecimalArray,PrimitiveArray(core arrays)VariantArray(core array)RLEArray,DeltaArray,FoRArray(fastlanes encodings)RunEndArray(runend encoding)ZigZagArray(zigzag encoding)ZstdArray(zstd encoding)PcoArray(pco encoding)DecimalBytePartsArray(decimal-byte-parts encoding)